1/20 



Q Start ^ 



102 

I / Picture Pair 
Model List 
(PPML) 



107 



Refine Models 








Robust Motion 
Modelling 


> 




Concatenation 


> 




Subsequence 
Partitioning 



101 

J 



103 

J 



7 J 



Reference 
Picture Model 
List (RFML) 



Yes 



Yes 



t A 



102 

V / Picture Pair 
^ Model List 
(PPML) 




105 



, Global ^ 
< C registration? > < \ User Selection . 



- ^User Selection ^ 



Global Registration 



104 



108 

J 



Reference 
Picture Model 
List (RPML) 



Picture Intensity 
Correction List 
(PICL) 



r 



109 



Global Intensity 
Correction 



Yes 




-< — ^User Selection ^ 



Compositing 



106 

J 



c 



Exit 



3 



FIG.1 



2/20 




FIG. 2 



3/20 



Q Start ^ 



Set eslimated 
number G of 
good vectors i 



Set Number of 
exact models 



Set picture number 
(PN) to first picture 
of sequence 



Set PN to next 
picture in 
sequence 



Yes 



301 

J 



Select minimal 
set of motion 
vectors 



302 
J 
— 



Construct exact 
model by fitting, 
minimal set 



303 

J. 




Yes 




310 




306 
— 




305 


y 


■^Yes 


Record 
G'^ smallest prediction 
error magnitude. 


Calculate 
prediction error of 
all motion vectors 
relative to this model 







Determine model 
with smallest 
G*^ prediction 
error magnitude 



309 



Save picture type 
- picture num. ref 
picUjre number and 

model parameters 



Calculate 
least-squares 
fit to selected 
vectors 



Select G motion 
vectors closest to 
chosen model 



308 

J 



Picture Pair 
Model List 
(PPML) 



311 

7J 



FIG. 3 



4/20 





i 



FIG. 4 



5/20 



For each I-picture or intra-coded P-picture which is reference by one of the preceeding B 
pictures: 

o look at B pictures whose backward motion model references the I-picture and identify the one 

which uses the most macroblocks, 
0 invert the backward motion model for that B-picture and assign it as a forward motion model 



Changed In this step 



502 




503 



In display order, concatenate all P-picture models where they exist to their reference picture 
models 



Changod in earlier stops 



Changed in tfi/s step 



505 




In display order, concatenate the forward models of all pictures (except those with identity 
forward models or no forward models or whose reference pictures have have identity forward or 
no forward models) with the forward models of their reference pictures 



> Changed In eariier steps 



Changed in Ws step 



B BI BBPBBPBBPJBB I BBPBBPBBP^ 

i' <!: — ^ — 1 1 4L — ^JU 1 1 4L — ^ — 1 1— -jL — ^JU Ijl — ^ — 1 1 — ^JL,^ — 1 1 — ^JJL 1 1 — ^ 



In display order, concatenate the backward models of all pictures (except those with identity 
forward models or no forward models or whose reference pictures have have identity forward 
models or no forward models) with the forward models of their reference pictures. 



— * Changed h} 



Changed in this «(flp 



For each B picture where both forward and backward models exist we have two models. For any 
B picture with both a backward and a forward motion models which reference the same picture 
make the forward model parameters equal to the weighted average of the forward and backward 
model parameters where the weighting is the number of motion blocks contributing to the model 
squared. 



Changed in earHer steps 



Changed in this step 



^3 ! 



Backward models remain but are ignored in subsequent processing 

FIG. 5 



6/20 



601 



Reference 
^ Picture Model List ; 
(RPML) 



Q start ^ 

Set picture number 
(PN) to sequence first 
picture number 



I 





Get reference picture 




number (RPN) of PN 




+ 




Set Subsequence 




number (SN) to 0. 




t 




Set Subsequence SN 






ref picture (SRP) to RPN 






t 






Set Subsequence SN 




— -<- 


start picture to PN. 



602 



603 



604 



605 



Increment picture 
number. PN 



Get RPN of picture , 
number PN 



JQ 



Set Subsequence SN 
end picture to PN-1, 



Increment 
Subsequence 
Number SN 



609 




Yes 



610 



Set Subsequence SN 
ref picture (SRP) to 
RPN 



611 



Set Subsequence SN 
start picture to PN. 



612 

J 



607 



-608 n 




Set Subsequence SN 
end picture to PN-1 



FIG. 6 



7/20 



Q start ^ 



Picture Pair 
Models List 

(PPML) 




Supplement Picture 
Pair Model List 



701 



Set Picture Pair Index 
(PPI) to first pair 



Get estimated picture 
pair model EPPM for 
picture pair PPI 




f 


Get subset of 'edge' 
points (SEP) In first 
picture P1 of PPI 




f 



702 



703 



Optimise EPPM to 
match pixel values of 

SEP in P1 to pixel 
values of SEP mapped 

through EPPM in 
second picture P2 of PPI 



I 



Increment PPI 



704 



Save refined model 
parameters for 
picture pair PPI 



705 




Yes 



c5d 



FIG. 7 



8/20 



( Start J ) 



807 



808 



809 



Sei Minimum picture 
separation (MPS) 



Set Minimum picture 
overlap (MPO) 



Set source picture 
number (SPN) to first 
picture 



801 



802 



803 



Gel reference model 
parameters for picture 
SPN 



Set destinalion picture 
number (DPN) to 
picture SPN+1 



804 



805 



821 



Increment SPN 



Set BPO=0 



806 



Get reference model 
parameters 
for picture DPN 




f 


Estimate picture pair 
modej (PPM) from 
models for SPN & DPN 




r 


Estimate 
overlap PO 


picture 
from PPM 



817 




820 



Yes 




r 



818 



810 



811 



812 




set Best 


Picture 




Overlap (BPO) to PO 








Set Best Destination 




Picture BOP to DPN 










BPO>MPO? 



Sat BPO=0 



Yes 




Add (SPN. BDP. PPM) 
data lo Picture Pair 
Model List (PPML) 







~1 

1-820 



819 



-815 



Add (SPN. BDR PPM) 
data to Picture Pair 
Model List (PPML) 



813 





-814 



Yes 



FIG. 8 



9/20 



C 



Start 



3 



Picture Pair 
Models List 
(PPML) 



Set Picture Pair Index 
(PPI)to index of first 
Picture Pair Model 
(PPM) In PPML 




r 




Get First Picture (FP) 
and Second Picture 
(SP) indices from for 
picture pair PPI 




r 



901 



Make Connected Picture 
Pairs List (CPPL) with 
all indices In PPML of 

picture pairs connect to 
FP or SP 



Reference 
Picture Model 
List (RPML) 



902 



Calculate new reference 
picture models for FP and 

SP which minimise the 
summed error in the 

picture pairs in CPPL 



Set PPI to next model 
in cyclical order 



903 




^Convergence? ^ 

Yes T L 904 



FIG. 9 



10/20 




FIG.10 



11/20 



Q start ^ 



Make seed point list 
(SPL) from key points 
in subsequence 
pictures 



J 



1301 



Generate FIFO source 
point queue (SPQ) 
for flooding 



1302 

J 



Initialise SPQ 
with seed points 
from SPL 



1303 

u 



Initialise flood map 
(FM) with seed indices 
of sources in SPQ 



1304 

U 



Set source point (SP) 
to first point in SPQ 



1305 

J 



1306 




SP \ 
overwritten 
, in FM? , 



Make a list of 
neighbouring points 
(LNP>ofSP in FM 



Set neighbour point 
(NP) to first point 
in LNP 



r 

1312 



1307 

J 



1308 

J 




Yes 



Set NP to next 
point in LNP 



> > 






r 

1310 


Sel seed index of 
point NP to of that 
of SP in FM 






1311 


Add FM point NP 
as nev/ source lo 
end of SPQ 




Yes More 
^ points in 



Delete source point 
SP from SPQ 



1314 



1316 





Use floodmap to 
make mosaic 




r 



FIG.11 



12/20 



start ^ 



1210 

tJ 



1201 

'J 



Picture Intensity 
Correction List 
(PICL) 



Y Select 7 
\ Compositor / 



1202 

J 

Flood Sequential 
^ ^ Connpositor' 



Create Compositor 
Object 





Picture Intensity 
Correction List 
(PICL) 



Initialise Compositor 



r 

1208 



I 



1207 



Flood Composite 
Pictures 



Reference 
Picture Model 
List (RPML) 



r 

1211 



Create Compositor 
Object 



r 

1204 



Initialise Compositor 



Sequential Composite 
Pictures 



Display or 
save 
Mosaic 



1205 




Reference 
Picture Model 
List (RPML) 




F1G.12 



13/20 



/ Reference 
Picture Model , 
List(RPML) / 



1300 



Start 



Make seed point list 
(SPL) from key points 
in subsequence 
pictures 



1301 

u 



Generate FIFO source 
point queue {SPQ) 
for Flooding 



1302 



J 



Initialise SPQ 
with seed points 
from SPL 



1303 



J 



Initialise flood map 
(FM) with seed indices 
of sources in SPQ 



1304 



r 

1312 



Set sourcG point (SP) 
to first point in SPQ 



1305 

J 



1306 




SP \Ves 
overwritten > ^ 
, in FM? 



Make a list of 
neighbouring points 
(LNP) of SP in FM 



Set neighbour point 
(NP) to first point 
in LNP' 



1307 

J 



1308 

J 




Set NP to next 
po/nl In LNP 









r 

1310 


Set seed index of 
point NP to of that 
ofSPin FM 


> 




1311 


Add FM point NP 
as new source to 
end of SPQ 




1316- 



Delete source point 
SP from SPQ 



1314 





1313 



Yes 



No^ 




Use floodmap to 


make mosaic 


1 





1315 

J 



FIG.13 



14/20 



1400 




Q start 3 



Raster scan Flood Map; 
save firsl occurrence of 

each seed index In 
Image Flood Seed List 



1401 



Generate FIFO primary 1 
source point queue I — 
(PSPQ) 







Initialise PSPQ 
with seed points 
from IFSL 





1403 



J 



Sort SSPQ in stored _^ 
picture order '"^ 



1404 



Generate FIFO 
secondary source point 
queue (SSPQ) 



1405 

J 



Initialise Mosaic Image 
and reset flooded flags 
in Flood Map 



1406 

U 



r 

1414 



Move next point in 
PSPQ to SSPQ 



1407 

J 



Set source point (SP) 
to first point in SSPQ 



1408 

J 



SgI flooded flag of SP 

in FM\ Set Mosaic 
Image pixel using SP 



1409 

J 



Make a list of unflooded 
neighbouring points 
(LNP)ofSP in FM 



Set neighbour point 
(NP) to first point 
inLNP 




1410 

J 



1411 

J 



Set NP to next 
point in LNP 



1413 



Add FM point NP 
as new source to 
end ot SSPQ 




Delete source point 
SP r«5m SSPQ 




FjG.14 



15/20 



Start j 



Set picture number 
(PN) to first picture of 
subsequence 



Reference 
Picture Model 
List (RPML) 



Get picture bounding 
box (PBB) for picture 
PN in mosaic 



1501 

J 



Set mosaic pixer(MP) 
index to first pixel 
in PBB 



Map MP location into 
picture PN coordinates 



get picture PN pixel 
value (PPV) at 
mapped location 




1503 



Accumulate data for 
pixel using chosen 
accunnulator 



T 



1505 
J 



1506 



Increment Picture 
Number PN 



Increment MP index to 
next pixel within PBB 




Picture 
Intensity 
Correction List 
(PICL) 



1507 




1508 



Use accumulator to 
make mosaic 




f 



J 



FIG.15 



16/20 . 



-1610 




Object in 
picture m 



FIG.16 



17/20 



r 



1710 



Object in 
picture n 



1720 



Object in 
picture m 



FIG.17 



18/20 



Object in 
picture n ~^ 




1810 



1 



Object in 
pictures 



FIG.18 



19/20 



Object in 
picture /7"^ 




Object in 
picture m 



FIG.19 



20/20 



2090 



computer system 2000 

\ 



video display 



208 



2045 



2055 

\ 




video 




storage 




I/O 


interface 




device 




interface 



2030 



2040 



processor 



memory 



Z 



I/O 
interface 



/ 

computer 2020 



7 

2050 



2060 



keyboard 



2010 



mouse 2015 



FIG. 20 



